TrackIQ Comparison Report

Generated at 2026-02-22T03:33:34.068050+00:00

Executive Summary

Overall winner: MiniCluster. MiniCluster won 1 of 5 comparable metrics. Largest deltas: throughput_samples_per_sec (+inf%), latency_p50_ms (+0.00%), latency_p95_ms (+0.00%). No regressions exceeded 5.0%. Consistency findings: none.

latency_p50_ms: +0.00%latency_p95_ms: +0.00%

How To Read This Report

Comparable metrics: 5Missing/non-comparable: 9Flagged regressions: 0Consistency findings: 0

Metric comparison: use this table for direct value deltas and winners.
Normalized deltas: positive values favor Result B; negative values favor Result A.
Confidence matrix: prioritize strong-confidence metrics for release decisions.
Consistency analysis: variance regressions can reveal stragglers even when averages look stable.

AutoPerfPyResult A

Tool: autoperfpy 1.0

Platform: unknown (Windows 11)

Framework: pytorch 2.5.1+cu121

Workload: html_report_data (inference)

Timestamp: 2026-02-22T02:22:55.013502+00:00

MiniClusterResult B

Tool: minicluster 0.1.0

Platform: CPU (Windows 11)

Framework: pytorch 2.5.1+cu121

Workload: distributed_training_validation (training)

Timestamp: 2026-02-21T04:05:37.557945

Metric Comparison

Metric	AutoPerfPy	MiniCluster	Abs Delta	% Delta	Winner
communication_overhead_percent	N/A	N/A	N/A	N/A	N/A
decode_tpt_ms	N/A	N/A	N/A	N/A	N/A
energy_per_step_joules	N/A	3.9279	N/A	N/A	N/A
latency_p50_ms	0.0000	0.0000	0.0000	+0.00%	tie
latency_p95_ms	0.0000	0.0000	0.0000	+0.00%	tie
latency_p99_ms	0.0000	0.0000	0.0000	+0.00%	tie
memory_utilization_percent	0.0000	0.0000	0.0000	+0.00%	tie
performance_per_watt	N/A	8.1625	N/A	N/A	N/A
power_consumption_watts	N/A	24.8406	N/A	N/A	N/A
scaling_efficiency_pct	N/A	N/A	N/A	N/A	N/A
temperature_celsius	N/A	36.6756	N/A	N/A	N/A
throughput_samples_per_sec	0.0000	202.7609	202.7609	inf	MiniCluster
tokens_per_sec	22.0401	N/A	N/A	N/A	N/A
ttft_ms	N/A	N/A	N/A	N/A	N/A

Visual Overview

Consolidated graph views for quick comparison validation.

Top Normalized Deltas

Positive favors MiniCluster; negative favors AutoPerfPy.

Metric Family Deltas

Family-level signed summary of normalized deltas.

Winner Distribution

Metric-level outcome share across comparable metrics.

Confidence Distribution

Data-strength breakdown for metric conclusions.

Normalized Metric Deltas

Positive values indicate an advantage for MiniCluster; negative values favor AutoPerfPy.

Metric	Family	Direction	Raw Delta %	Normalized Delta %	Advantage
latency_p50_ms	latency	low	+0.00%	-0.00%	tie
latency_p95_ms	latency	low	+0.00%	-0.00%	tie
latency_p99_ms	latency	low	+0.00%	-0.00%	tie
memory_utilization_percent	memory	context	+0.00%	+0.00%	context

Metric Family Delta Waterfall

Family-level mean of normalized metric deltas (context-only metrics excluded).

Family	Metrics	Normalized Delta %	Visual	Winner
latency	3	+0.00%		tie

Metric Availability Confidence Matrix

Confidence is based on metric availability in both results.

Metric	Family	AutoPerfPy Available	MiniCluster Available	Direction	Confidence
communication_overhead_percent	communication	no	no	low	none
decode_tpt_ms	other	no	no	low	none
energy_per_step_joules	efficiency	no	yes	low	insufficient
latency_p50_ms	latency	yes	yes	low	strong
latency_p95_ms	latency	yes	yes	low	strong
latency_p99_ms	latency	yes	yes	low	strong
memory_utilization_percent	memory	yes	yes	context	strong
performance_per_watt	performance	no	yes	high	insufficient
power_consumption_watts	efficiency	no	yes	low	insufficient
scaling_efficiency_pct	other	no	no	high	none
temperature_celsius	other	no	yes	high	insufficient
throughput_samples_per_sec	performance	yes	yes	high	strong
tokens_per_sec	other	yes	no	high	insufficient
ttft_ms	other	no	no	low	none

Consistency Analysis

No consistency regressions detected or all-reduce step data was unavailable.

Platform Comparison

Hardware differs: unknown vs CPU.